In conventional systems, the manner in which system memory is allocated to processes executed by a system is not adaptive to the overall load on the system. In such conventional systems, the maximum memory that can be allocated to such processes is a fixed amount and cannot be changed, even when the system has a lot of unused system memory. This can result in underutilization of the memory that is physically available on the data processing system.
For example, a data processing system may execute multiple processes in parallel, with each of the multiple processes being configured to perform a particular operation, such as a sort operation. Generally, the maximum amount of memory that can be allocated to each of these sorter processes (or sorter) is fixed. For example, in a particular scenario, this maximum amount may be set to 64 MB. This number is fixed irrespective of how many sorter processes are executing at a time and also irrespective of the total memory usage for the data processing system executing the sorter processes. Since each sorter process can only be allocated up to the fixed amount of memory (e.g., 64 MB), in situations where the number of data items input to each sorter process for sorting is high, when the memory consumed by a sorter process reaches its allocation limit, the sorter has to write the sorted data to disk (also referred to as spilling to disk), in order to free up memory to sort more input data. Finally, each batch of the sorted data written to disk has to be read back from the desk and merged into the final sorted output data. The spilling to disk and reading back from disk is very costly from a time perspective, and very resource-intensive. Instead, it is preferable that each sorter can perform and complete its sorting operation fully in-memory (i.e., in system memory) without having to perform spilling.
In the example given above, the maximum memory that can be allocated to a sorter process cannot be changed even though there may be a large amount of unused and available system memory on the data processing system executing the sorter processes. This results in inefficient underutilization of the available memory on the data processing system and also makes the overall sorting process more resource and time-consuming due to more frequent spilling to disk.
Further, simply increasing the threshold for the maximum memory that can be allocated to a sorter process is not an adequate solution because the overall available memory for the data processing system is fixed and has to be shared between the sorter processes and other processes executed by the system. Setting this maximum memory threshold unnecessarily high could result in other processes being starved of essential memory resources and this may also adversely impact the overall functioning of the data processing system.